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depth withffi a predebainiynad range, is carried out 
|>a8ed on the deptfi intoirmation. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention: 

The present insertion relates to methods for creating an image for a three-dimensional display (hereinafter referred 
to as a 3-D cfisplay), for calculating depth information, and for image processing using the depth information. The 
method for creating an ffnage for a 3-D display particularly relates to a method for creating a Pseudo-viewffinder image 
shot l>y a multi-eye camera (a stereo image) from a viewfinder image shot tjy a monocular camera. The method for cal- 
culating depth informaSon retertes to a method for obtaining a cfistance t)etween an object and a viewer, which is appli- 
cable to practice the method for aeating an mnage for a 3-D display. The method for image processing using the depth 
information relates to appTications including the creation of an image for a 3-D display and the suspending of creation 
oranim^fora3-D(isplay,anenhanced display, or ttie like. 

Description of the Prior Art: 

[1] Creation of an Image for a 3-D Display 

In fields related to television techrvques for creating a 3-D image (a pseudo stereoscopic vision) through the detec- 
tion of a movement of a 2-D motion image have been knaf»wn. One typical example d such a technique is a 3-0<fisplay 
errvfoymg a «me difference methiod. the princ^ of wfiich will be described wfth reference to Rgs. 1 to 3. 

In a scene where an object moves from left to right whBe the badvound stays stiD. as shown in Rg. 1, by repro- 
ducing respective nii^es for rigW and left eyes (hereinafter respecti^ 

to have a predelernwied lapse of time between them, as shown in Rg. 2, a parallax e is caused/^ 
parallax" or "a binocular cfisparity" is defined as an ancMar different 

right and left eyes, respectively. In Fig. 1 . since a viewer perceives the car as being doser than ttie bac tegrowd due to 
parallax, a pseudo stereoscopic vision is achieved. When the object the car in this case, moves in the oppoGfte (Sec- 
tion, respective irre^ieeshodd be reproduced such thaL«»ja^ 
left eye by a predeterniried ^e. contrary to ttie exannpie ste^ 

JP Publication Na Sho SS^6240 (jsctoses axfisjpifay^aparBte for a stereoecGpic image using ctepth wior nriatoi. 
in which the apparatus receives only an image signal shot from one b^ 

signals from mult^>le *ections and a signal containing the depth infomiation of an obiect. so as to reproduce within the 
sqsparatus the original viewfinder image shot from rmiltiple directions. The purpose of the apparatus is to r educe a trans- 
mission l>andwidth. The apparatus incorporates a variable delay drcwt for causing a time delay whOe contrd&ig the 
extent thereof accoidbig to deptfi information. The Hme delav results kt a paraOax. Accordng to an output si^ from 
the droBt. image si^ials are reproc&iced for ngjhA and Mt ^es. In this way. a pseudo sta^eoscopic image b cfisplayed. 
This publication ffi^^^j^^gag as a pr^erred enrtxxftnent dthe disdosed apparatus. (1) a device for cfisptaymg a pseixJo 
stereoscopic m)aQB tor a viewer by respectively supplying right and left eye images to two CffP s. which OTe gftiated 
fbrmmg a predetenrtned angle with a half nwror. and 9) a device lor displaying a pseudo stereoscopic image for a 
viewer even if the viewer moves in a horizontal c&ection. using a lentioiar lens fixed to a display scr^. 

Howei^, the iteive apparatus works on the concftion that deptfi irlormation is sMPplled extehiaBy. Tbercfcre, If it 
ortfy receives a 20nwtion inriage,tfie apparatus cannot create a pseudo stereoscope 

JP Application Lad-Open Ma Hei 7-59119 rfsocisctoses an apparatus for creating a pseudo stereoscopic Image 

based on a 2-D motion iiTttige. The apparatus convns" « <*«*e<^ 
2-Dmotfonmage. «iBd^cfecuMDrdelayniocHherarijJhtoraleftimag^ 

causes a paraBax. TI»iypBca^ 

dteptay (hS^D), which is a glasses type tfisptey lor supplying different images to ri^ arxl left eyes: Through the HM), 
a viewer c^ri see a pceudo stereoscopic tmage. 

In tts BFr^"*fc* howwar, sroe the extatt of <telay determined accoitfing to the maspitude of a ftofion vector, 
sfftf dbi&A rrwving at high speed appears to be cipser to fte viewer, residing in an irmaturd ste irc^^^ijjy which 
is ciscord^ to «ie effecfive tfistance between the viewer (or the canrm^ lind the obiect (fiwilii^i^^ ^^ : 

JP Laid<)pen Appfoation Na Sho 60-263594^50 cfsdoses an appmtus far cfisplaying a ( ^Kfo ^ era ioecnpic 
image uang a time durance method, in which the appw^ 
so that they are sewafternaevety via shiaertfasses for «^ field, as fwshu^ 
and left eyes. This «3fication further disctos« a r^^ 

affiSence betwee^^^ moves at lew speed. However, since tt^ apparatus also does 

irt operate bise*fed#tfi 
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-PIXEL" (No 128) a magazine, issued on May 1 . 1993 describes in pages 97 to 102 a pseudo stereoscopic im^e 
system tS"ng dUliSmat'n. In the system, an object isfirst dispteyed as a gray^e image ^J^J^'^l 
fieTco^SS^ndsto the depth, and then based on the gray le^el. the appropriate parallax .s cal<^«»«*r,*^'^°L?^^ 
nt!^r<JS^S? S that rijt and left images are aeated to be seen via shutter glasses. However, the perspectve 
image is manually aeated and a technique for automating the creation 6 not disclosed. ^^^r.^^ 

Mfltional Piljlication No Hei 4-504333 (WO88A)4804) discloses a method tor achieving a pseudo stereoscopic 
image ^^ de^^^n. The method 

*A riKriSdar«s deoth intormalion. so as to provide each of the areas with a parallax, and for creaftr^a pseuooster 

^ * ^^""""^ *^ "^"^ me supply « not 

Sl-eye camera), or using a monocular camera while moving It Several aspe^ of this tech^ 
iSSrt ^iSed^isO Pictire Coding Symposium of Japan (PCSJ90).- for example, on page 57. 

[2] Creation of Depth Information 

Computer V«on would enable detectfon of the depth of an object *f «*«''f^ 

whi*jS2l on 2-D molfon infom«lion. suitable images are not always suppjed for the '^^^J^.^^^ 
Hont^uS^en with unsuitable images supplied, serious enore are likely to be caused. That is. if 
! S^i^f,^s^iL«table ima^ and then used tor the creation of a stereosccpic image, rt may bequrte Bkely 
S^SrS^e^^;^^^^*^ -i' be ««-ural. i.e.. exh«ting such anomalies as a person m .he d«.ance 

''''^TZt^'Si^^!!^^ understanding of a corresponding rebtionship 

kJLS, *^Ss^lS^ Sf^SS^P ApplkatforH^Sopen Na Hei 7-71940 (which corresponds to 

camera, so as to estimate the position of the pointer ine in actual space (Ihe ^Dwortd)- *2^^ii2«S^«^^Imte 
S^dSTon a camera whie moving it so as to obtain ils sequential v|e^^ 

^a^iSSerS^point on the sequential viewf inder images, and thereby estimatang the posAon ol each charactenste 
point in actual space. 

[3] An Image Processing Method Using Depth Information 

A method for controlling the movement of a robot using depth information is known. ^J^*^^^^^^^^^ 
ter^SrTnSStocJ^ 

*"S.'"«roS£s.a,:sss"^^^ 

stereoscopto image has scarcely been proposed. 

■«4l IMMARY r 7f T"P iMVgNTION 

The fW object Of the presert inventfon relates to the creation Of an image for a 3-D c^)^^^ 
aho««Tl 1 to dS*« the oSecl of the ^ 

1 An acfl«testereoBOODic image based on depth informalfon IS not created Instead, a me^^ 
iiilS^lSSSS^^ent Of mc^eS^ent Further, since a parallax needs to^«Je- ^^^^ 
STSW^iriZdmSer^horizonlalmovemertrt 
conslitutes alurvtonental I 



2.AsHisnolautomated.lheprocessforob.aininBdep«,infam»rtionf««ia2-^ 
process. TlwB. a stereoscopic image cannot be output in real time ifx*i iifoi* 011^ 

Therefora fie fist object of the present invention te to aeate an acojrale stereoscopic imape. based ondepth 

^""r^^lS^S^acccrdingto^^^ 
imagi bSd cn^hieh an imago*» a 3^) displw isc«^ 
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to a technical fiekd relating to an image display. According to one aspect of the present invention, depth information is 
obtained through the following processes: that is. the movement of a 2-D motion image is detected: a relatave 3-D move- 
ment between the scene and the shooting viewpoint of the 2-D motion image is calciilated: and relative distanc^ from 
the shooting viewpoint to the respective image parts of the 2-D motion image are calculated, based on the relative 3-D 
movement and the movements of the respective image parts. Based on the thus obtained depth infbmiaton. a pseudo 
stereoscopic image is created. ^„ 

This aspect of the present invention can be differently described as a depth being obtained through the following 
Dfocesses that is a plurality of viewfinder frames (hereinafter referred to as frames) are selected from a 2-D motion 
iJraae to be processed; and a relative positional retaSonshp in the actual 3 

identiied based on a 2-D positional disptecentwrt between the frames. In other words, in order to determine the depth. 
3-D movements of the respective image parts are calculated based on the 2-D positional displacemenl. b^ on which 
Dosilional coordinates of the respective iniage parts in the 3-D world are calculated, accord 
gl^^. A frame is a unit lor image processing, that Is. a concept inducing a frame pidure or a field picture m MPEG. 

""^S^Sng a 2-D motion image, a pluranty of viewfinder frames are hereinafter referred to as Mifferent-time 
frames." as they are shot at different times. (In the following description of a multi-eye camera, a plurality of frames 
which are SiiTiultaneously shot are referred to as isame-tinw frames.-) A positional (f^ 

r^ed to as -a 2-D positional displacwnert.- In this aspect of the present invention, where different-time frames are 
<fscussed -a 2-D posHionaldisplaoenMnrnieans a change caused along wWi a lapse of time. »wt is. a rnoven^ 
((>! the contrary. -a 2-0 positional displacemenr of same^inie frames means a positional dM^ 

^ *TlSUcond object o#thepresenllni«rtion relates to the calculation of depth infonTiation.asdeecri3ed intheabove 
El Thatis.thesecondolJiectofthepresertinv«ntioni8toproposeamelhodforobtainingaooireclcorres^^ 
toBiM) among a piuraity of iinages, so as to calculate accurate depth Wb^^ 

onrapriate tor the cafcutaBon. and far <Sscontinuing the calculation of deptii WormaBon when any toconvanance 
occurs, such as coiJd cause an unnatural pseudo stereoscopic image to be created. Further, the present Biven^ 
to propose methods far effectively determining corresponding and characteristic pa 

'^'taoidwtoacS^ScI^ second 

between them are selected fioifia 2-D molfan image, so thatdepth infomialion is oWainedfiomthetwo frames. A ccctd- 
ino to this aspect ol the invertioni l is poBsijIe to obtain a good ealoialio^ 
n«y faalitate the cakailalioa A judganart as to whether frames have a^ 

based on the extent or variance of movement of a c»«racteristic point. ^ . -, 

According to another aspect of the Bwention. with a representative poirt provided in a reference frame, t^^ 
ity of Bnages is evaluated between a iinage area induding a characteristic point in the otiier frame (hereina^ 
to as an obiect frame), and an image area including the representative point in the reference frame. A cterac tensbc 
DOir« is a candidate lor a conesponcing point subject to an evaluation, the candidate being arWra^ 
the relattve positional acceplabOity between the characteristic point and ^ 
is. a iudaemert is made as to whether the relative positional relationshf> between the characterisfc 

dUiaSrfeticpoirt is reasonabte or acceptable wtm 

relafionsh^ between the representative point and the other representative point. respecSvely corespondw g to the 
characteristic pointa VWhan both evakiaions reaA to a fawordUe score, the chara^ 
as a oxresponding poiiil of the representative point SUJsequentl)^ a best poi^ 

ualionB yield the bestHesMt bymoving^ane ujnespond i ngpoint within a predetonnined search area. »Wte asswi«g 
that al the other coire^ohdinri»lntB arefixed ^ method hereinafter being lefened to as a fixed «earehing method). 
The best positioa wlichitos been touid dwtog the search, is detemiined as a new position of toe<orreyondingpo*it 
An corresponding poirta art sequentiaiy subjected to this eeerch and the positional change. Afterwards deplh wtor- 
mation is obtained based on a positional ref uB on sN p belMveen a representative point and coiresponcSng port, toe 
corresponding point ha*^ been dbtrined through the above mer*oned series 0* process 

Oonventionalyi the siri^^ has been evaluated by block matching or the lite. tothiSBwenlwn. on toe 

otoerhand by indlidjno an itiaifeiiri exaluaBon on the relative positional evaiualioa the oui ieH JUiufciB reiahonshp 

between frames caii^» ni*i^i«rtely^«^^ 

Accordvig toOffS*Bp^1*iflfe present li^ o#«ie mnges is waiuated by Uocfc meMmo wtach 

is motified such that Ifte sini^ eoirectly evBiuatod to be highest 

tested, retaacfless of awottogcciTd l tion s Ow w 'ttf'M ^ '*^^ 

certain color deflection tends to occur due to characteristic dfferenoes of a plwality of cameras. As to <ifferert-time 
frames, the same problan wffl arise due to changing weather from time to time, as this causes a changem bns^^ 

(tf a viewfinder in^ft Mtercwaclien te made to 8ali««uch problems, the si^ 

\imi^m ^^^SBim i»euie»mtm^ concept far judging «ie acceptehfliy of relafcw poeiiions. 
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ton which is preHdetermined depending on a distance between the reference and object frames. That s. when the d.s 

tanr*» ifi laroer a laraer correction limitation is set accordingly. 

A ^^on for'off-setting a change in br^htness is disclosed in JP "-^^ 
corr«*^ is applicable only to cases of ladng-out or fadngnn (a eonsistert cha^^^ 

^TS^iSnSer aspect of the invention, d^ infonnation Is obtained through the foUo-nng P~c^- ^^f; 

I^^ml i^mes is selected because such a point is considered to be accurately traced. 
^SJrSS,SS^^Si^^S^the^«ent inven^^ 

■^i^^^t^a^^'STpresent inventlco. depth ^matlcn is ^'"^f 

ih-t WT^Zlity of reoresenttrtve points are provided in a reference image: a plurality of corresponding pc«^ of the 

««^2^^SSSmriSte eadirS«entative point has a sole corresponding point, efimmatmo the POsaM"^Jhe 
^^^^^<S^'^<^uSZ^^^ positior^. This ideal^ sole coaesponding pent « the true 

SSS^SfW POBibons other lhanihat Of ihe true cone^ng p«nt and 
jL^^SCTdMrtTwilh' oosilK* depth 

^ . Zl imaoft w« be created acconSng to the changed depth information. An excessweiy large paf<u» 
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appearance of the frame. The correction may be made by uniformly cutting oH a peripheral part of the frame at a certain 
width, so as to achieve a desired shape of the image frame. 

According to a further aspect of the invention, in a method where image processing is carried out for a 2-D Image 
according to its depth infonrotion. an image area subject to the image processing is determined, based on the depth 
5 information. With this arrangement it is possible to separate an object or to change the scale of an object a certain dis- 
tance from a viewer. 

According to the further aspect of the invention, in a method where image processing is can-ied out on a 2-D mage 
according to fts depth information, images with viewpoints at a plurality of points on a hypothetical moving path, where 
a shooting point of the 2-D Image is hypothelically moved, are created for use as a slow motion image, based on the 

10 depth irifoiiiiation. 

It is to l)e noted that, accordffig to the present invention, a viewfinder image seen from a different point may be cre- 
ated accortfing to depth informalion. A positional displacement of respective image parts, which will be caused accom- 
panying a change in the view point are cakajlated based on depth information, so that a viewfinder image is re<:reated 
so as to correspond to the positional displacements caused. When a viewpoint is changed in height for exanr^jle, a cis- 
ts ptecement (ttie extent <rf translalion and rotation movem en ts) of the ot^ect (respective image parts) can be caicuteted 
based on ttie cfistance l>y which the camera has nrx>ved and the depth information, so that a desired viewfinder image 
will be created based on the calculation result. 

BRIEF DESCRIPTION OF T HE DRAWINGS 

20 

Fig. 1 is a viewfinder ffnage where an object nwves from left to rigWwhi^ 
Fig. 2 shows reproductions of ri(^ and left ffnages havvig a 
Fig. 3 shows a parallax caused due to the lapse of tme ol Fig. 2. 

Fig. 4 shows mam stages for the creation of an imi^iefor a 3-D display according to Embodiment 1. 
25 Fig. 5 is a flowchart for *ie detection of a corresponding reiationsh^ between viewftf^ 
Fig. 6 shows representative points provided in a relererK^e frame t. 
Fig. 7 shows block m a te hin g. 

Fig. Sisaconcef^moctelwherethevalueof El is indicated for a tentative corresponding poM PTOi) in a per- 
pencficular cfrection. 

30 Fig. 9 shows a retefibhsNp between a representative point and its oonresponcfing point determined at 812. 

Fig. lOis an expfahatory dtagrm regarding a princple of evaluation of a relafive position ol oorrespondmg points. 

Fig. 11 shows a result olinprovement processing conducted on cancfidates for coiT^^ 

Fig. 12 shows a relationst^ between movements of Point P on a saeen and a 3-D space. 

Fig, 13 is an explanatory c&^am regarding a princq^le erf determining 3-D coordinates of Point P. t>ased on the 3- 
35 D movement of a camera and the movement of Point P on a screen. 

Fig. 14 shows representative pootts each gn^en an actual numeric value. 

Fig. 15 shows a parafiax given acoorcfing to depth information. 

Fig. 16 shows and left images created from Frame t 

Fig. 1 7 shows a rm-feieartanstormation with respect to a parallax. 
40 Fig. 18 shows an example of a hardware structure for practictfig Embodiment l. 

Fig. 19 is a monoctvonrtepic^eshGwffig a viewfinder hrageffiFram^ 

Fig.20isarnonochroniepic*ireshowirioavlewffoderiinageinFwrier. 

Fig. 21 is a rnorKX*wonw picture of Firanrw t overlaid wi^ 

Fig. 2ZfB€L nvxKxhrome picture showing Mial po si tiof g of corresporxfing points in Frame t'. 
45 Fig. 23 is a monocfyorrw pictire Showing correspoiKirig points 

Fig. 24 is a monodrorw p icture embody^ a (ray-scale imy. 

Fig. 25 is a nrwnodwome picture of a right image aeated acoorefing to depth Mormatoi. 

Fig. as is a monbU M ume picture of a left knage created aooortfaig to deptti i nfo r ma tion. 

Fig. 27 shows main stages for me creatfon off an infiage for a 3-0 cfisplay aocorrt 
so Fig. 28 shows a selectibn oteria «^ naspect^ a^cha^ point which is infrocfoced in Entxximent 3. 

Fia 29 shows a uir^r esp biiting r ei ttlionsh » ^ m original viewTnder mage and one re<rerted so as to be seen 

from a changed vieMfxM. 

Fig. 30 shows a c^K'^'ft sp orKl^ reiationsh^ of an original viewfinder image and one re-created so as to be seen 
from a chmged v iewpo i rC 
ss Fia3l8howsantfnagewihaparte9>anded. 

Fig. 32 shows an irnage wfth a house separated from the irnage in Fig. 29. 
Fig. 33 shows a structure ot a stereosccapic inage (feplay apparatic in Ent^^ 
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ncQ r.RlPTinN OF THE PREFERRED FMRnOIMENTS 

Preferred embodiments of the present embodiment will next be described with reference to the accompanying 
drawings. In Ent)odimenls 1 to 4. an apparatus outputs as a final image an image for a 3-D di^ay (a pseudo stereo- 
scopic image), while in EnixxJiments 5 to 7, it outputs an image for a 2-D display (an ordinary 2-D image). 

In Emlx)diments 1 and 2, the apparatus initially receives a viewfinder image shot on a monocular «mera white in 
Embodiment 3 rt receives a viewfinder image shot on a multi-eye camera (a stereo image). 

conespond to Embodiments 5. 6 and 7. respectively, except that the fomier outputs an image for a 3-D display and ttie 
latterfoTa 2-D display. Embodiment 8 relates to a dispteying method, in which unique conditions for a display apparatus 
are considered when dispteying a pseudo stereoscopic image. 

Emtiodimentl. 

Fig 4 Shows main stages tor the creation of an image tor a 3-D dsplay according to Embodiment 1 . Up to the third 
stage, the content of the method for calculating depth irAxmation according to the present invention will become appar- 

^ In Ent>odiment 1 . an image tor a 3-D display is created based on an image tor a 2-D dsplay through Stages 1 to 3 
tor amdyzing an image tora^D display, and Stage4tor creating an imagefora3-Ddisptey Respect 

beoutined. 

[Stage 1] Extraction of 2-D Motion Information 

Information about the movement of an object sho^ in a *W is fi^ 

is 2-D at this stage. That is. coofdirwtBS are 
!>s scTAAn wil be exoressed by means of 2-D coordinates. ^ . 

T^Tto^i^terstaiSl the movemert Of the object a c^ 
detected. A viewfinder image at Time t is designated as a reference frame (hereinafter re^^ed » ^^f® 
a^ewf inder image at Tone r is designated as an object frame (hereinafter retorted ID as "Fmn* n^Uy^laj^^ 
JS^esJLe points are pre^rovided. so that corresponding points of the 
30 f=r^r.^amestandrcor»titutedili8rert4imelrame6with«^ 
in terms Of frame sequer«. Stage l is characlerizedby the tact lt«d 

horizontal movement of an ot^ect but also from movements in any diredwn. Hereinafter m this f^""*"^' « fjj <^ 
are defined as ttme. and a frame is defined as a unit constituting a viewfinder image in general, but is not liinrted to a 
particular frame of a television picture receiver, which comprises 525 scanning Hnes. or ^ screen crfa persona c«^ 
35 Z. which comprises 640 x 480 pixels or the like. Alternatively, representative points may be provded in not only Frame 
t but also both Frames t and f. 

[Stage 2] Calculation of 3-D Motionlriormation 

After Identifying the 245 movement of the object mformalfon about an actual 3-D 
lated as 3-D motion informatfon. Tbe 3-D iTwtion is expressed by « parameters: 
tioa TOs calcUalion is rrade based on a pluraity of pairs of representative and corresporiding p« 

[Stage 3] Acquisiten of Depth Mormation 

Identification of the actual mo«mentd the object would define a lelative positional '>f|^J^ 
objecls at differert timea Further, identification of •* relationship cooW prov^ 
respective parts (hereinafter referred to respediye hnage parts). 

50 [Stage 4] Creation of Image 

A parallax is determfcwdbased on thedeplh information, so as to create rigW and 1^ 
mined ttich that a ctoser Object has a laroer pamllax. Sinoe respective iw 

and left images should be ereated such that the 
55 becleaftyunderstoodlhalthefollowinofactsaredillerentftomeacholherands^ 

thal1h€m«rtfon infomalfon can be ert^ 

inwNdi a parallax is provided is imited to a horizortal direct 

viewing the object 
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Respective stages in Embodiment 1 have been outOned above. In the followring. they will be further described in 
detail. 

[Stage 1] Extraction of 2-D Motion Wormation 

Fig. 5 is a flowchart for detection of a corresponding relationship between viewfinder Image frames, respective 
steps of wtwt> will neirt be described one by one. 

(Step 10) ProiAJing a Representative Point in Rame t 

As showr> in Fig. 6. representative points are provided in a Reference Frame t. In Rg. 6. Rame t is cfivnded aito 
everv 8 X 8 pixels by overlaying it with a yid. and representative points are provided at every crossi^ 

^ iSSenSicular^ of the grid. The representative point of the ^om le« j;";*^^ 
eiiressed asF?i j): a coire^jondfrig point of Pt(iJ) « 
IS are expressed, if required, as Ptfi,Dx and PtCij)y. respectively. ^ ^ ^ A.o««tr««o 

Arepresentative poirt inay be provided not only at a crossing part but at also any desired pa 
case. aB pixels may be individually designated as independert representative points. 
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(Step 11) Setting a Corresponcfing Point Candidate Area 



so 



Taking an exanple of PK6.4) in Fig. 6.. an area which may possWy include Pr(6.4) is pre-determnrfbased on the 
assumpli«i fiat R-(6. 4) be positioned in the vicinity of Pt(6. 4) except lor a drastic movemeritof a viewfinder irwso 
I^^TS^ a predeteLi«S1imilation. in B 

Pr(6.^ is assuned as eadsSno in an area of 100 X 60 piMte in the vicinity of Pl(6.4). 
25 Step 11 cai also be modified as foHows: 

1 Whenaviewfinderimagemovesrelativelydrastically.twoframesadiacertintermsoffr^ 
nined as Rames t and r so as to mininwe the endart el a change in the position of represei^^ 
Rainestand r. and Ihusihe risk of disptacino the can«pondhig poirt from the assumed a^^ 
sWe to assume the whole Inage area as a corresponding poirt candidaie area. The risk* 
sponcinB point from ihe assumed area due to a teige nwvenrwnt of the 
ttie vohine of caladalion is resuttanUy mcreased. 

2 m the above, a corresponding poirt candidate area has been detenrtned based on a simple assumption ttwt 
R-(6.4) be located in the vicinity of Pt(6.4). However, when the movemwit of Pt(6.4) is tra^ 
frames, a corresponding point candidate area can be detennined on the extensK)n of the moj^^ 
method is particularly advantageous in Bmiling such an area in the case ol a viewfinder image with a r^ 

slant movemmL 

40 (S 12) Ctfotetion of Noi>-SBT«gity in a Ckaiiesponding Poirt Candidate Area 

Theposi8onofacoiT«poh«*,gpoirtls8peciiedintheoonespo«^ 
arises wheivihe viewfinda image moves considentfjiy ^ 
moves only by a small extort, it is dfficult to extract motion infomiatk^ 

45 inctodedininiormaionisincraased. » ^_,^^,,.s^^j^y,rMimf 

to owJer to prevent such a problera.Tni»rfepre^(«lected such fiatFiani^ are set«partfcom each^ottwr 

by some odert. to other words, alter conducling stalislical analysis to the extent of change^ 

T»ne r is selected such that the magnitude of changes or the variance of the ttdent 

nxned valua Mernalively; Tme t- may be selected such that the total sum of or the variance 

than a predetermined nu«*«r of chatecterislic poirts^ "^J^^^TIH^^ «SSh2Sr 

thatm^totte abovecdriilipnsis notf^^^ 

raalion) is cisconlinued and^toitoadL an Input viewfindeir image wil be cutout intact or aH image parts oi toe viewnnaer 

imue are dtoplayed'as if liiiMiiD a uhionn deplh. . , »-_^».s» 

toftisstop. to otdef ^^Bennne the pdeilion of a correspond poirt. non-smteity between Rawes tend t is 
oonputed by bkx* matehtoo melhod. That is. the total sum of squared dWerences of gi^-s^ 
te^^Ll between one btock having a certam poirt as Its carter irMJec^ 
another block including the represenft^point so as to detect a certan poirt p^ 
d etwn i n edasaoompulBdcuiie&pcndtoopOB*. 
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Fig. 7 shov^ block matching. In Ea*odiment 1 . nine pixels constitute one block with the central pixel as a repre- 
sent^poim of^^^ t including R(i j). whUe Btock 2 is provided on Frame X\ including Pf(i.i). that is a ten. 

tath^e^datJ^r^^^^ a pixel value of a pix^(x.y) at T.me t designated as n(x.y). the non-sim- 

ilarrty (hereinafter referred to as E1) is generally obtained from following Equaton 1 . 

E1= ls{lt(PtOj>+u.Pt(ij)y+v).|tXPt{ijlx^4J.Pr(ij^^^^^ [Equation 1] 

wherein two S's relate to u and v. Since u and v respectively take the values of 

u = -1,0.1 
V O.I. 0,1 

^ *«^«^P»T.i^ p 5«uared difference Of gray-scale level can be Obtain 

""'^/ai^^^^SS'Slg the valued '^'^'"flC?"* 
Q /;^t^n^^?StCndin9^ since K sho«s a steep peaK in non-sin.terity. In this way. correspond-ng 

points of all represertaSwe poinis are detennined. 
Step 12 can also tM modified as toilows. 

i In 4h«aha« a souared difference of gray-scale le«l has been calculated.^ 

l^-^ V^S^ noJSrilarity may be the total s«n of squared dWerenee of 
■n^.inousFi^ a Alternalively. the density of olher color spaces, such as an HVC 

ence of gray-scale level. s 
o in this steo. «ne pixels constilute one bk>cKthou^ H is preferable that 

a io un d 16 x 16 paste. 
(S13) Determination of an Initial Position of a Corresponding Point 

UptoStepl2.atentativecorrespondingpointhasbeendetermined.«io^^^ 

^ . ^ ' u*«rf«. rtr orViAR nf an cbiect may have been determined won saDSiacrory awAirauy. 

r^t^r^^ITsScSSfSSi^^S^ a representative point and its conesponding point, theco-re- 

Sto^S^ialwpieri^althooj^ corresponding points relating to chaiacteristic 

JSr^Ta^SSe^ST^ 

"'".^ep^raStSXSS^ 

m isiep lo i-^ the mnoeot of an Wtial posHion is wrtroduced, so that the mital position of 



i naj foved through repeated , 

The inrtia!posHioni6deterfn«ed.follo^ 



Au ...r^^nn noifite liMcii havtti>eeh determr^ %jo to Step 12 are ec^y processed m Slap 13. 

AU corresponcingpoinls wtiK»i^^ iJlt«i arrrenarded as their Mial posiion tor the sUbee- 

Positiors where aD corregporKf ng pomts btb now kxated are regaroea as meir ™»i 



so 1. 

quent processing. 



55 



^ ^^^iSSSSL^S^S^^SSSSUy have been detennined with satisiaclory aca«cy (hereinaf- 
ter r^e.^r.^^SSSSS S^-^e Ih-V -e ^ 

S lSn?e?h^^farott«r cowndrng poinis (hereinafter referred to ^"^-^-^^^S;*^?^^ 

S p^SSedetermned based on 1hp«i,ol 

Sto^^tocOTMatesfcracharacterislicp^^ 
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to coincide. In this specification, representative points of the corresponding points as a characteristic point are also 
referred to as a characteristic point 

(1) A corresponding point having a defffiite peak in the value of El in Step 12. 

Generally, such corresponding points are quite ia«ly to have been positioned with a high positional accuracy. 

(2) A corresponcfing point located in an area including many orthogonal edge components. 

Con^esponcfing points included in areas around edges of buildings are qiite likely to have been correctly posi- 
tkKied. 

(3) A corresponding poirt whose positkxi varies steac% from Ranriett^ 

The stes^finess may be understood as the consistency of a motion vector. Therelore. a correspondfeig point 
moving in a consisiem nwvtng Erection by a conststert diste^ 

be selected as a ch^acteristk; point Concretely speaking, a con^esponding point to be selected should have a 
motk>n vector wrfiose variance is tower tfwn a predeter 

been traced precisely anrxyig respective frames, and thus having heen judged as having a correct corresponding 
reJattonshp with its representative point However, when the camera has nxsved irregiiarty, the influence thereof 
must be oonsktered in the judgement. 

When a characterislk: point is determined, its position is used as an initial position, while the initial position of a non- 
characteristic pomt will be interpolated by using neighboring characteristic pointa In other words, since the posittonal 
accuracy of a non^haracteristic poirt d^emined up to Step 12 is tow. tt^^ 
metricaiy based on the neighborwg characteristic points with po^^ 

12 can be u3ized in fMoig a characteristiopoinft described In *ie above (3). _ _ 

In addrtion to tie abovennentioned nfietfiods based on the selection 
conespondkig posilon may be determmed by a dynamto progre^^ 

(S14) Inpioveniert Process for the F^wtton Ola Corresponding 

An equation is ntroduced to evtfuating positioned ac^^ 
tive positional acceptabiay tfwoagh fteraliwcakaila^ 
another equation is introduced for evaiuating accaptebility of a relalive posi^ 

points. The ©raluation results derived from both of the equattons are combined to inprove the positional accurrcy 

Referring to Fig. 10, the princple of relative posittonal evaluation vwll be descrft)ed. Fig. 10 shows correspondffig 
points. Taking Prfi,D as a center, the foltowing four con^esponding points are tocated adjacent thereto: 

PfO-i . D. wch-i. d. P. i+i) 

tt is reasondbty assigned that Pr(yl Is kxated around the center o# gravity of these ^ 
based on the experience that, even when respective image parts nwv^ 
tIaBy maintamed. This experience can be natheniaticaB^ 
ferentirf of Prfij). which is a funclton of I OTi jrte substenliany zero 

Therefore, wfth the center of gravity of ttie four poWs being expressed as {St (i.i)x. STClDy). Equation 2 

E2«{PtTi.»x-SrfaM?^<PtT^^ [Equation?! 

is obtained for evaluating relative positional aoceptebiily. With consideration d Equation 2 only, a oonrespondng point 
wifl be most lavor*ly positioned with the mirwnum E2 valua In o^ 

is ««tuated using the fuoitioncldistarioe between neighboring inra^ ^ 

^ «s step, evdafttion resite deWed ftrom^& 1 and 2 are combined with an ^ipropriate coupfoig factor k. 

Ther^e, a frwl eiiifibAfon equattoir^ can be eoqpressed « Equalton 3 

E«E1/N+kE2 [Equation 31 

wherein N is the nuni>er of pixels ffiduled m 

for the inprwemem of the relative positional acceplabWy, E is 

poirte. Then, after addmg all Fs into £E. the respective corresponding positions 

the virtue of IE. This conputation is repeated untila^^ 
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to a predetermined nuirtjer of iterations. That is. concretely speaking, any of the foBwing methods is practiced while 
moving respective conesponding points. 

(1) A Method Using an Euler-Ugrange Differential Equation 

When an Euler-Lagrange differential equation expresses IE taking an extremum (a relative minimum inttiis cas^. 
a corresDonding point is obtained by solving this Euler-Lagrange differential equatioa This is a known method, /^rd- 
ina to^method. a direction in which a corresponding point is to be moved for improvement from its inilal posibon is 
determined based on both gradient in respective blocks including a representative point and differential beWveen corre- 
sponding blocks, so that the corresponding point is gradually moved in the direction from its initial point until reaching a 
final sohJtion. 

(2) Rxed Searching Method 

In a corresponding point candidate area, a pohn is searched where the value of E of a correspond 
improved becomes minimized, and then newly set as a corresponding point. The fixed searching method «cteracter- 
istic in that the search is conducted for one corresponding point while others are keptfixed. The above mentioned proc- 
ess is repeated with respect to aJi corresponding points. 

(3) Hybrid Method 

According to the method (1). « is possible to positioo a coaesponding poirt with an a^ 
of less than a pixel, white acco«Sng to the method (2). with an accuracy in units of a pixel. Thwefor^ 
utifize both methods, that is. first applying the method (2) to obtain a correspondinQ relatfo 

units of a pixel, and then the method (1) to enhance the accuracy. ^^««^-«.„iho 
Exper^nents have shown that the method (2) provides a favorable solution n a shorter penod el tme than the 

method (1) used to obtain the same lewel of accuracy. 

Fia 11 shows the result of irijrowmert processing according to tWs step, which has been condu^ 

to canSdates far correspondinB points shown in Ffl. 9. Experiments have sh^ 

in a cotor image, with tie coupfing factor k at around 5 to 200. 1^ 9 and 1 1 show mode! results, th^ 

iments have proved «Ht iriproMenriBr« dose to the model results ¥«s realized. 

This step is characteristfc in that 2-0 motion Information can be extracted from the movement ol w 

directfon This is an advantage achieved through understanding of the movement of an object by introduang the oon- 
ceot of reoresentative and corresponding points. This advantage makes the present inventkxi applicable over a wider 
i^e. conned to a prior art in which a time difference has been determined through the detection of a honzontal 

movement 

Step 14 can be also modfied as follows: 

1 in obtaining E2. a center of gravity of eigW points may be detemined. the eigW points induding four poi^ 
onally tocated from the center, that is Pt(i.O in Fig. 10. as well as the four respectively tocatedup^ ^..^^TIl^ 
of. to the leftofandtotherighfofihe center. PreferaW y. the optimum combination of the points is determmed exper- 
imentally, as It depends on *ie Wnd o* 2-D image to be processed. 

2 Evaluatkxi by Equation 3 shoiid begin with a corresponding point whose evakialfonresuM of Equation 
favorable, because a drastic mprovemenl to such a corresponding poirt at an early stage B prefe^^ 

eraUy considered to have a laige error. 

3 For the improvement of the positional accuacy. geometrical infonnationshouW be util^^ 
representative points forn*ig an area with a geometrkai feature, such as a straifiW Bne. in Rai^ 
their a>rresponding points should be eoriectod so as to also lonn the same geometrical feah^ 
madefor reasons fhaimp«rtw»**i seen* to be alme in a viewfinder Image IsqMte ikBly tofan^ 

3-0 world as wee. «|;a iine*i the 3-0 world shoiJd form a Hne in Firame f as weB. Since th^ 

varies ansistanay atong a me. and because sudi a linear variation bevtaaj|^r^^ 

rection by the *ove menlfoned method wffl acWeve a signifkart inprovemert V^Whoul sueh^m 

final image nwy indude iwegularity in depth atong a Bne. thus possUy resiiling in an unnatural 3-D display. As 

alternative geometrical Inlonnatton. edges of an image area can be used. 

4 Further correspondHig points are oWamed with respert to other frames as weB. h this stag©^^^ 
points are'oi^y obtained in Fiane r wilh rwped to Firame t thoua*) it is possito to oWa^ 
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a third frame, or Frame r. so as to obtain averaged movements of respective image parts. This method is not for 
improving a relative positional accuracy of the coresponding points in Rame t'. txrt rather for statisticaDy determin- 
ing the movements of respective image parts. t)ased on the respective positions of conresponding points, which 
have l>een provided in nnany frames, and the respective times when the respective frames are shot 

5. When an insufficient nun4>er off characteristic points are estat)lished. the ongoing process is discontinued 
l>ecause it is quite unlikely that an accurate corresponding relationship will t>e obteined. 

[Stage 2] Calculation of 3-D Motion Information 

In Stage 1 . the 2-D movwients of respective image parts on a screen have been identified. In ^ge 2. 3-D move- 
ments thereof are calculated b^ed on the identified 2-D informatioa That is, since the 2-D movement in a viewfinder 
image is a projection of the actual 3-D movement of an object onto a plane, the original 3-D movement of the obfecl is 
calculated based on the positional retetionshp between representative and corresponding points in a viewfinder image. 

Movements of an object m the 3-D world can be generaUy descril>ed as a combination of translation and rotation 
movements. In tfie foBowing, a method for calodating a movement comprising translation movements only wiH be 
described first followed by an example of a generalized metiod. 

1. Translation Movennents Only 

Fig. 1 2 shows a correspond ng relationship between the movement of Point P on a screen and Hs actual movement 
in a 3-D space, m Rg. 12, the 2-D coordirmtas are expressed with a capital letter, whfle the 3-D coordinates are 
expressed witii a sttbH tetter, mwtiichx«ly axes are provided on t^ screen, white the z axis is in the 

depth direction. The dtelarice from tte viennioM and the screen 

As shown tti Rg. 12, P(X.Y) moves to P'pc.r) « the 2-D screen, whfle S(x.y.z) simultaneously movae to S(it, y*. z-) 

in the 3-D space. 

When the followkig equation is held 

(3^.y'.z')«{xy.z)+(aj3.c) 
since the screen is placed a cfetance cl 1 from the vtewer, X Y x* and y* ca^ 

X=xte. Y= yfz 

x=x'/z\ r=//z' 
By solving the above, the following is introduced. 

X=(Xz+a)/(z4C) 

Y=(YZ4by(Z4C) 

Therefore, with z eimmated. Equation 4 is obtained. 

(a-Xt) (V-Y) « (b-rc) OC-X) [Ec^iation41 

Since Bc^jation 4 is ej^sressed in terms c# nrwvenienlB on the s^ 
(b) and(c)accordtfigtothe irifornriationoWainedinSt^1. 

that is k times larger moves at a speed k times Nflfter to a place awey k times further, the vatue d k (a sc ale fa ctor) 
cannot be detem^ned, the ratfoiotvahies of M,^^ and (c) to one another can be sotely cto*ainej|^*^iematfoaly 
speaiQiKI. even i three pairslii;^^ and (x^y) era given, since the rank of a coefficient matrix of thisisinniitaneous 
equation is as tow as two, (a). (B). ind (c) cannot be detormtoed as real values but only as relative vakna. Therefore, 
in tNs ^soe. tie of W is non«*zed to one; eo as to express tie values of (a) and (b) as a ralto against (c* 

because a ratio IB sulfkaentlyuBabte in the subeequertpiooessi^ _ ^ 

An afternrtive solution with respect to translaBon movenrients is as folkJ^ 

as Equation 5. 

e-Cta-X-cXr-YHb- Yc)pC.X)} ^ [Equation 5) 
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e={(Y-Y)a-(X--X)b-{XY'-X'Y)cl}' 

T>,pn the total sum ze of all (e)s regarding all corresponding relationships between representative and corresponding 

Smsrcrulat;S,2JLT.he;e^ 

the value of 2e. 

d(re)/da=0 . [Equation 6] 
d(Xe)/db=0 [Equation 7] 

d(Xe)/dc=0 [EquafionSl 
More concretely speaking. Equations 6 to 8 are respectively developed into Equations 9 to 1 1 . 

aECr-Y) 2-b2()C-X)Cr-Y)-cS(Y--Y)(Xr-)CY)=0 [Equation 9] 

^S()C-X)(r-Y)+bZ(X--X)=^*cr(X-X){XY'-)CY)=0 lEquationiOJ 

-a£(r-Y)(XY'-X^-»b3:(X'-X)(XY--X'Y)4C£(XY-XY) =^=0 [Equation 1 H 

2. Movenients Including Rotations 

Movements including both transtelion and rotation can be expressed by means of three '*S«f''«*r",*;j^ 
z ax^^SSS^^S*^ rotalk« angles, 
rr^^rf^^teSon Rotation ancles can be expressed by means d an Eulenan angle or a ronprtchmelh^ 

^IIJS th^STeS^^esLX nexttobe obtained. However, as explained above, since a scale iactor 
c^n^b^^S^^^^ ^^^iabtes to one another is sde^ 

^^^Tp^ a movement when given five pai« of ^^P^eserrtalive andoon^^ 

?^^i8^St«eSS^d^^ on1heselectionofthepairs.the content <>» T^^rL 
ified T,^^^ on a linear transtomation in some cases. However. H is known 
^id^eS^Ju* cases, gro^ 

vol.26. Na6. 714/720 (1990). 

[Stage 3] Acquisition of Deplh Information 

Relative extent of the 3-D movemerte of the respective image parts «enHW "j^^^J^^ \ 

tion vector (a,b,c) as toaows: 

(xy.z>R(x.y.zHaAc) 

Theirtvetsetransfonmtk>naftte 
movement of the camera. 

(x.y.2)-R'{(x'./.z>(ab.c)) [Equation 12] 

Refemng to Rg. 13. the prindpte for obtaining S-D coordinates of Point P. r^''^^ 
and 2^^ZnJi of Pbint P on a screen wBl be explained. This P^^P^i* ^^J^-^ Sfli^^S^ 
in which, when viewing the dbedion of Point P from two separate pomts. 

12 Mrt^SroitSa^S^ 

"^^S^^ er. Which are fom^d by the direction ^^^^^^^ 
tively, are tewwn and the directioo in *hich the camera moves and te 
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be to obtain 3-D coordinates of Point S. Based on the 3-D coordinates of Point S, the respective image components 
can be provided with their depth information. 

tt is to t>e noted that, as descnt>ed abcve, due to the normalization of (c) as 1 , the obtained 3-D ooorcOnates of Point 
S are having been expanded or compressed by a uniform ratio. Howe/er since it is uniformly expanded or compressed 
5 as a whole, the depth intormation retains a correct relative positional relationship among respective image parts. 

In the above mentioned processing at this stage, it is necessary to consider enors wtvch have been caused up to 
the previous stage. In other words, due to such errors. Lines Lt arvJ Lt* often do not cross each other as a result of cal- 
Guiation. To cope with such a prot)lem. a point is provided at the middle of a line connecting points on Lines Lt and Lt' 
where the lines are closest to each other, so that a z coordinate of such a point will t>e approximately designated as a 
10 depth of Point S. This process will next t>e de8crit)ed using an expression. 

When the direction vectc^ of Lines Lt and Lt' are respectively expressed as (u.v.w) and (u\v\wO. txsth Lines L arxJ 
L' can k>e expressed as following Equation 13. using parameters of aand p (real number). 

Lt : (x.y.zHa(u.v,w) [Equation 13] 

15 

LT : (x'y.z>p(u\V.wO 

Therefore, when an enor (e) is expressed as the foUcwing 

20 e-{(x+pu)-{x+au')) ^+{(y+pv)-(/+av')} ^+{(z+pw)-{z*+aW)} ^ 

ttie valuesof aand p which minimize the value of (e) are obtained using the expressions: deAja^O and de/dp^O. In 
words, by solving the e^jations of 

25 {U^-w^+w^)a-{uu*4Vv'4W¥OP^<x-x>**(y-y7v+(z-z>»M) 

(U' ^+w' ^)p-(uu'4W*+w¥Oa4<x-x^*+(y-y')v'+(z-z')w*=0 

tie values of a and p are detenmied. so that the depth of F^o^ 

30 

{(z-M«i^zVpw^}/2 

Especially in the case that the error (e) is zero, the (z) coordirtate of the midpoint coincides with that of the aossing point 
of Lines Ltand Lt*. 

35 As an altemative method. Lines Lt and LT are both perspectively projected onto the screen of Frame t. so as to 
obtain the (z) coordinate of the closest point of Lines Lt and Lf . In this approach. Line Lt is projected as one point on 
the screen, while Line Lf is one line in general. Witi Line Lf expressed as Equation 13. the (x) and coorcfinates of 
9ie points on the projected Line Lf on the screen are expressed as Ec^jatkMis 1 4 and 1 5 by dn^icfing (x) and (y) oooidi- 
nates of the points on Line Lf in the 3-D space by ttieir (z) coordinates, respectively. 

40 

x«4(x*^pu'V(zVi-pw7 [Equation 14] 

y«4(y'+pv7/(zVpw^ [Ec^jation 15| 



wherein (f) is an actual dsstarice from the viewpoint to the eoreen of Fran^ 

from Equations 14 cmd 15. Line Lf after beingpfciected 9«ereinafter referred to as Li) can be specified asi 

Kx^Hny+tn«0 



so wher«n Imt^'-wV . m»-wy-u'z\ rMjy V^. The closest point to be detected is a foot of a perpendfcular from ttiere^ 
resentative point Pt to Line U (here^iaftfir retenred to as Pdnt D), that is^ a point where a fine di n w n tom the represent- 
ative poirtf Pt meets Line U 6&« to form a right angia md the ooonSnatas of Point D are expre sse d as following 

Fquatiiift 16. 

55 x-(in^X-lgvtonY)/(k^4m^) [Equation 16] 

y«(k^Y-mn-tonX)/(k^4m^) 
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Assuming that the original point on Une Lf in the 3-D space, which corresponds to Poirrt D. « ^^^^aj^t^^P™'!^ 
^"T" E can be detected by substituting Equation 16 into Equation 14 to obtain p. and further substrtuting the 
obtained value of p into Equation 13. Since p is expressed as 

p = (xz'-fxO/(fu'-xw*). 

by s^ituting this expression into Equation 13. the (z) coordinate of Point E. that is z". is detemined as the following: 

z"=z'+w'(x2'4xV(fu-xw') 

"""XnlhtS^J.'vSTte'^^veTet errors caused in image processing, the compUed value |snot re.^ 
t^^irn^STe^^TeJ^^PointSexistebehind thecal 

becaieenne neg-n^ representative points dose by with a pos*we value. 

"'irrC^"c?::srnS^^^ 

resoeSr^esentafive points as an actual numerical valua Rg. 14 shows ^epr^errtata>«pomte^ J^Si* 
g^;^aS^!i^l Sue For exampte. the depth of Pt(2.3) and Pt-(4.3) are respectn^hr 100 and 200. the latter 
actually being located twice iffi far away as the farmer. 

so [Stage4] Creation of Image 

A parallax is detemwied acco«ling to the depth Wormation. which has been acquired In Stage 3. so as to create 
^«h.o^u*kn«« this staoe. a fcffther image Is lobe pr^ _^ 

In Fig^S, wNbh "^^^^^^^ „ rL , ^ provided on a viewf inder image shot by the camera under 

"^R:S\*S^i::J^trL^S^ are respec^vely pteced as shown in F«. I^J- R and L^ejjj^ 
resi)^y«S2SSi to right and^ a viewer. Then. St(2.3) and SK4.3) ^'If*^ "Lf^L J^^ 

L^S^SfrS.jS?rS^^JS« on a display of lenticular lenstypeorthelite.whichisdis- 
SS^inTJSS?SiS^!^^ it is possible to obtain a good ^ . 

!n li^ InSlHr^ 1hert2«»scopic image may be generated for a desired part only whwh has been s^arated 
In this en*odiment. '"j J^son is located 5 meters from the camera, with mountains m 

*^ ^^^Jl^^^^S^^MB^W area, so that right and left images can be aeated with respect to 

r S^SS'cSlS^^^^t^ blanKorUng the P^*^^^^ 
only me area cuni»r »• •» " cfonAR im^ to Staoe 3 in the number of a wewf inder image frame to be used. Up 

vSL in ills order. Tl« image part of the pereon. closest to the viewer, exh^ 
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45 1. having a largest cfisplacenient to the left in a ri€M 

2. having a largest cfisplacemert to the rigtrt in a lell imag^^ 



^ . ^ . ..rw*«fc>«d that the above (1) is a situation where the viewer sees the pereon from a point which is 

fcm»4 IriK case. • Is n«»«w B 
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1 . Non-Linear Transformation 

As is shown in Fig. 16. some oJ the divided parts are transformed into a trapezoid. Awidely-used linear trar^orma- 
tion such as an aff ine transformation, however, cannot be applied to such transtormation. Therefore, in order to trans- 
form a part with four vertexes into a trapezoid, a non-linear transformation, such as a perspective transtormation. is 
applied. 

2. Linear Transformation 

In the transformation into a trapezoid, provided that a part with tour vertexes is first d^ 

ina three vertexes. a linear transformation can be applied with respect to such a part ^ 

As a result of a horizontal displacement of respective image parts through the above mentioned tra.»lu..kwaHin, me 
peripheral edge of the Image may become un««n. m Rg. 1 6. the bottom parts of right and left images are <«^lanci 
with respect to each other, and accordingly the peripheral edges of the displaced pans become crwtod. 
Therefore, by adding piwis to such a recess, the shape of the Image 6 corrected ba* into lis ongmai shape (a rectan- 

^ Tllntep!^ m image part that faBs on the added pixels Is determined while referring to the depth c« those dose 
to the paels. or in other ways The images on the added pixels can be seen only by one the ej^ 
phenomenon and arises in an area close to a window frame when people took ootsKle m 
nrted that this correction can also be made tjy deleting redundant pixels, wtii* 

lively, the peripheral part of the image is uniformly cut off by a certain widlh. With this correction, wespedive of the 
selection among the above methods, it is possible to maintain a natural displ ^ 

In this stage, a parallax is deternined according to a depth, though the partdlax IS preferably fu^ 

the foBowing reasons. 

1. Eye^tigue 

in the aiwve exaripl* ft Is not desiiabte for even a person dosest to me viewer to be giww 
deoth because an image perceived to be excessively frontward from the screen wouW cause fatigue 
^ Accordihg to a report in -NW«i Bectronics- (April 4; 1988. pail), it Is mo*deslial)le for respectwa meeparts 
to be given a depm in a range between 0.2m to 2m. when a display is positioned 50 cm fro^ 

2. Personal Preference 

Some people prefer a dose image to be displayed mudi doser and a distart image much farmer. whUe omers pr^ 
fer the oppostte. 

3. Processing Capacity 
H an image areas constituling a far background, such as a mountain. areiSsplayed as « havi^ 

the volume of data to be processed can be reduced. 

Because of the foregdng reasons, in this stage, the foltowing functionB far transforming a depm or a paraiax are 

{^jpied as requested. 
45 1. Depm Tri m sfor ma lion IHBWlion 

A depth is dreclly sU)jed to eimer a Inear or a nar»<near transformation! Thsi i^ 
is a depm. and a paralto i^ fesultantly changed. For eoiample. as for a vi«linder image compnsing rmagepartewam 
depmsintherangeof lato 10a({a)beinganartHtiary«kje). medepmsof mere^ed^^ 
mjUfieed by ten. so that all depths fan in the range dl llhrto 100a Tlis depfc lranBlormalion finction is advantageous 

foraviewfinderint»ge wim anexceBBiv«lysmaidep^ • 

Altematively. when the depm is in tie range «f 0 to lOOa. the depm m«be compressed, fw eacanryla to the range 
such as 25a to 7Sa wim SOa as me origin of transfarmaion. AS a further alternal^ 

toortessman 20a. or emaltoor nwiefhaiTlOdda^B^ a un!farm20a or 1000a oe pm. 

mtNscasThoU-er.asaresultoftheirt^ 
i^Tthrt is 1000a and 20b. becoftie <fsa*itinuous. and thus fonn an w 
order to soh« this problem, a non-lnear transfomwtlon is applied such that images snwothh^ 

lower IMtatian values. In this eoenple. me fotkMnng transtormatian should be made; 
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z->a/l1 +exp{-(x-0.5a)/aT)}+zO 
wherein (2) is the original depth. 20 = 20a. a = 1000a - 20a = 980a. and T = 4. 

2. Parallax Transformation Function 

A oarallax is subject to a linear or a non-linear transformation. That is. after a parallax is calculated based on a 
.ep:r:rns.ormi.the^ 

Rg. 17 ^ « ~"^t^^^aTS^ a Viewpoint i to the Une L Tr« depth of Point 

vided on a central line L and R»ntB« at tt^t^^ ahalfof the parallax) is set as shown in Fig. 17. 

^"S^S^^^^i'ereTe^CS^c^^S;^^^ eSTs^ to he trammed to a point 

which satisfies the tallowing Equation 1 7. that is Point S 

e=e/2 lEquationIT] 

■n, «f Point S- is exDressed with a segment SU A series of processes in connecBon with the transformation will 

^r-XStSS tSa^^.^ d^-*ned using the dep^ SB. according to the relationship o. 

e«atan(SB) 

S'B is next determined according to the relationsNp of 

and efficiency. 
Fmhodiment 2. 

The optimum apparatus for practicing Eirtxx&nertt 1 will bedescribed. 
finder image frames tor oetecoon w cones^ a^or-r- .^...^.^ «r *h« likA is used lor block matching. 

oriniirf*SterSL*cl# image is retrie«d*^ 

signallv an inage output circuit 36. to be supplied 10 an unniustrated deplay. 
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24 For a normal 2-D display, the supplied viewfmder images will be displayed irttact, or alternatively the viewfinder 
images stored in the frame memory 24 are sequentially read out therefrom for display. For a 3-D display, a plurality of 
frames on a viewfinder image are read out from the frame memory 24. so that depth information of the object will be 
obtained from the read-out frames. ««th the corresponding point detection circuit 26 and the motion detection circuit 30. 
Subsequently, the image aeation circuit 32 aeates right and left images according to the depth information. 
The instruction input section 34 can be structured as foBows so as to achi»e the following functions. 

1. Structured as a Control Knob 

The sense of depth on the created image can be adjusted so as to satisfy the personal preferences of a user by 
varying the sense of depth through scaling depth with a control hnob. The rotation of the knob may be ac^usted in 
advance such that the minimized sense of depth will provide a 2-D display. 

2. Structired as a Pointing Device 

(1) The sense of depth IS adjustable in units of image parts. For example, when the person in Fig. 16 isdesiredto 
be displayed much closer, a poiming dewce. such as a mouse, is used to poirt to the peiw. and then di^^ 

a result the irnage creation drcuit 32 transforms the depth information of the person for use in an e^ 

by avmg a wider paraBax. The effect of this atfiustmert will become more signHicant if the dsplay area of the 
selerted Item is also changed together with the change of the sense of depttt Conoetely speaW 
sense of depth, the Cfaplay area of the selected item w* be expanded tour times. 

(2) A viewf inder image seen from a tfifferent point can be created Since the depth infomtMtion is available, by des- 
ignating a shooting poirt (a vle»«pdiil) of the object through diddng with a nwjs 

a cAaialion of translation and rotelion movemente of the respecfive inf«ge parts, the nm^^ 
acconmanying the change of the viewpdnL Therefore; a viewfinder image to be seen from a differed 
created. In Fig. 16. far exanvia a viewfmder image to be seen after the viewpoint is changed in heifltit or by 
advwdng or withdrawing the camsre. «an be re^yeated. Further, since the depth infarmaHon of the r ecreated 
viewfinder image can be computad through the calculation, its 3-0 display can be mainiainad m a goodxo«»on 
with the parallax changed accoidaig to the newly COTputed depth BTformai^ 
ferent point wai be furttw deeotoed later in Cmboiftnent 5. 

In the following, the results of experiments. In wtnch the apparatus of the present sivention is mourted in a wiork 

station. wB be descril)ed with reference to the drawings. 

Figs. 19 to 26 show image creation procedures with the apparatus according to the presort inventcn. Each 01 1^ 

drawings is a BW picture on a display which conrprises an area including about 640 X 480 piMls. 

Figs. 19 and 20 are viewfinder images in Frames t and r. respectively, exhibiting some movements between them 
due to a positional drtference of the camera Fig. 21 shows the same viewfinderimage of Fig. 19. witha grid ovetteid 
and representative points provided. Fig. 22 shows the same viewfinder inrttge of Fig. 20. with cor^ 
their initial position, in WW* the initial position is set at the temporary best point The ten^ 
through bio* matching. WW* has been conducted, beginning with a characteristic point with r 

X 16 pixels w»h a represertativepcint at te center. „ 

Fig. 23 *ows inproved positions of con^espondhig poirts. presenting a signaicant Bnpi^^ 
resJt of Equation 3 in Enfaodimenll lor considering a positional relationship beNieen correspond 

Fig. 24 expresses depth mfomiaBon at a gray levet where a lighter level represeniB a snaler 
from •» drawing th« depth Wbrinafion hasi*^ oonsideratole aecura^- --y: . 

Rgs. 25 »id 26 »e right and left nra^ee. f eepecliveljii which are created based on dep Ci wfo nrnhon. ^ doser 
object a can i n this eanple. is shomi to have a liwder pandtex. and is thus g^ 

As deecrtoed above, with the presert waratus. i is possiblo to autornatically practice the rne«^ 
1 of the present inventiort Further, the appUcaHcin of haidware far block matohin o contifcute s consideiafaly to tie 
iirvjrovement of pncBsmW^omtmia*^»» Bxaamg time required witt the applicalfanof «f™[«^ 

The present appaiirtw eto bb ell«eiivily«^^ ir» a product by attaching an add-on-eaid havng the stnieture 
as shown in Fig. 18 to a peisbi« odiiij*t«r or a work staiian. or pra^nstaS ng a draa ha wing the s kucture as shown 
Fig. 18 to a trievision receiver, a videopUver or the Hia Fulher. ly contjining the praeent apparatus^ a camera, it 
is also possiUe to shoot an object sepanrtely fh)m ite sumwndinBs in o^ 
from ifSerert points, so as to prcAice a catato^ cortoning 3-D pktures of the object Witti tte way 
measuremert by means of a laser, infrared rays, or sijpersonic waves, which have been a»iv^^ 

no tonger necessary. 
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Embodiment 3. 

Contrary to Embodiments 1 and 2. in wtnich a monocular camera shoots an object, in Embodiment 3^a multi-eye 
camera svstem is used to captire a stereo view! inder image. The captured stereo viewfinder image b used for the ae- 
ation of an image for a 3-D dsplay. In the fdlowng. a method for such an image creation is described mainly in v.ew of 

*''^'S*i^^.SJSg?ttiough which an imagefor a 3-D display is created. The difference from the stages in 
Fig. 4 of Embodiment 1 is the foflowing. 

1. At Stage 1 in Embodiment 3, displacemeftt information is e)ctracted instead of the motion information in Embod- 

""^ile different-time frames are processed in Embodimem 1. same-time frames are mainly procewed h«e in 
En*X)diment 3. Between frames shot at the same time, the mo^^ent of an object cannot be defined. THus, mtor- 
mation about the object displacement b^een such frames is extracted instead. 

becaiee the distance between cameras is already known as shown in Fig. 13 and depth information can be 
obtained accoiding to the provaple of triangulation using the distance. 

When inaccuracy could be caused with respect to the relative positional rela*^ 
of a multi-eye camera system, it is desirable to use seHcaltoatioo to correct such inaccuracy at 2. Mettate for 
^^Sb^^edescrSdinrelerencessuchas-Self-Calibcati^ 

S^rt S^mation Processino Society of Japan. Vtol. 31. NaS (1990) pp.650-659. JP Lad-Open Na He. 2-138671 
as and JP Laid-open fto. Hei 2-138672. 

Stages 1 to 3 of Entxxfiment 3 win next be described. 

[Stage 1] Extraction of 2-D ttsplaoement Wonnation 

In addrtion to the substitufion of motion informabon mth dsplacement i^"^^";^^^*^^^ 
Fmmes1and2 wliich are respectively shot by Camera 1 and 2 at Time t In En*x)dimert 3. it s possWe^ 

SS^ie^^^ni^JSy two^ame^ wNd, are shot at the s^l^. that ^J'^^^^^J^ 
when shooting using a multi-eye camera, the viewfinder image captured may be a sbll «T«ge. Stage 1 is further diHerent 
from Embodiment 1 as follows. 

(1) m Step 11 of Embodiment 1 (Setting a Corresponding Point Candidate Area), the amowit of caJcul^ is 
reduced wm the appropriate selection of different-time frames or the HmHation of a correspondmg point caidj^ 
area. are conducted based on the intensity or the trails of the movemert 
mert 3. on the other hand, a diflerert method from that in Embodimem 1 is employed as descnbed in the 

40 to Hmit a corresponding point candidate area for the same p»*po6e. ^. ^ , 

Tis Su^Sa nmS-eye camera is positioned horizontally, as te usually the case. Y ooo^«^^^^ 
coordinate) of coiresponcing points in frames shot by carneras of the mutti-eye camera system are sU>^^ 

the same as one another. TkWngthisinto consideration, as wen as errorsdue to irnage ^ 
Sc^n corresponding pow caiKJidale area can be Bmfted to a horizontaly 
assu.;edthat^mesrand2areshotatr»r»tandFr^ 

a positional difference of the representative poinis between Frames 1 ' and 2' is "^"^ 
stirring poirt candidate areas in l=«uiies 1 and 2 be set so as to have the san^ dH^ 

^Nveenea* other. In other wools, the oorresponcSng poirt 

the regions, the tfiflerence between wWch is aboi* X 

so 
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(2) Although statistical anrfysis is mtroduoed tor a slow movement in Step 12 of Embodmenl 1 (CataAdion rf nol>- 
silT«larity in the area far candidates tor a oorrespontf ng pdnt), this analysis 

(3) Similaily to Step 12 in Eiitwd i mertl. block matchirig is introduced in determira^ 
poinis n &i*xximent 3. Howe-er. in Entwimert 3. biased b^ 

Sock matcKng in some cases, such as when the multi-eye camera system to be used « c«^^ 

with different chaiacteristies. 1^ exanple. if Camera 2 tends to produce moreW^^ 

color density of Fhmie2slioiM have its blue conponents (B) subtracted to a certan extern (^ 

tion constant ae) befare inleigofeio Mock matcMna VWlhoul such an adjustment, there b a nsk thad the meanmo 
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of E3 tor contiing El and E2 may become invalidated. An example will be taken where a color density is expressed 
in red. green and blue spaces. In such a case, not only blue (B). but also red (R) and green (G) should undergo 
such an adjustment through siirtraction of color deflection constants aR and oq. respectively. Note that the biased 
t)lock matching evaluates the similarity, based on a squared difference of gray-scale level. This means that the sim- 
ilarity can be treated as a distance in the color space, which is the same metric as is used for relative positional 
acceptdality of viewf inder images. Therefore, the similarity and the acceptability can be combined together and can 
be used tor the matching evaluation. 

Referring to Fig. 7 and b£©ed on Equation 1 . biased block matching will be desort)ed using equatkKis. Pt(i j) in 
EntxxJiment 1 is denoted as PI and P2 respectively corresponding to Frames 1 and 2, and lt(i j) is as II and 12. 
Since Equation 1 can be simplified to be expressed as Equation 18. Equation 18 can be used k\ normal bk)ck 
matching with respect to a gary-scate image. 

E1»zx{l1(PiX4ii,P1y+v)-l2(P2x+u.P2y+v)}^ [Equation 18] 

On the other hand, biased block matching is represented by following Equation 19. which is a nKXfificalion of 
Equation 18. 

E1«i:r{l1(P1x+u.P1y+v)-C(P2x+u,P2y+v)-a}^ lEqualion ISJ 

Fbracok>riniage,v«thabeinganyoneofaR. €XQ.andaB. El is calculated for all viewfinder images in all RGB spaces, 
so as to obtain the total thweof. thati6,ElR+ElQ-^ElB. which is used in btock mafching. For sinplicily. Equation 
19 can be expressed as Equation 20. with llmd 12 representing l1(P1x+4J. Ply+v) and l2(P2x+u, P2y+v) . respec- 
tively., 

E1=y:i:(l1-l2-a)^ [Equation 20] 

wherein II and II are furKtions of u «tfid v, respectively, aid a is a constant. 

The optBTwm value of a is ofatemd next Since Cameras 1 and 2 shoo* the same obrect viewfmder knages 
captured by both cameras should comprise sUbstartally the sanrie owrtent except for the cfisplacemerte of tie 
respective vnage parts. In other wonts, tie nrwe sirate the characteristics of ttie cameras aie, ttie sra^ertie 
value of El in Equation 20 beoome&iBased on ttiis fact it is knowrr thai a shodd be a ¥^ which can miqirnzB 
thev^ueoTEI. Since Equation 20 can be eiqpressed as Equation 21, 

E1=Zi:{(l1-l2)^.2a(l1-l2)+a^} [Equation 21] 

-£2(11-12) ^-2aEX(l1 '\2}*1.Tol ^ 

provided that the total nuntjer of pixels in a block is N. Equation 2 1 is further expressed as Equation 22. tor ££ 1 » N 

El*Z2<l1-l2)^-2a2:2;(l1-l2)+Na^ [Equation 221 

Therefore, since 

dE1/da»-2££(l1 -l2)+2Na 

held, the value of El is nmnrBzed when Equation 23 is held. 

cLm(SE^-12)VH [Equation 23] 

Since o can be understood as an average d»Brence vahie in color densft^ 
weas off block imtehirig. a subsMfim 

E1«XI:(I1-I2)MX2:(I1-I2)}^/N [Ecv«tion2fl 
Th^efora ft is conckjded ttat Equation 24 is ised for biased bk)ck n^ 

is assumed that Cam«ras1 and 2 shoctexactfylhesanFieobl^ becomes zero. Onthe otter hand, tie 

value of E2 also becomes sitetantiediy zero. Therefore, it is unde^^ 

imtffig an initiitf error causedhy the judgement as to iiMmilariiy of viewfmder images. Afterannls. tfie best makftng 
wig be segchedthpii#fife«y^ ^ 1- 
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It is to be noted thai a color density other than an RGB density, such as an HVC densrty. nnay also ^ applied 
without a problem tor block matching. Moreover, block matching may be carried out based on a color difference^ 
S a rSSal difference, instead of a squared difference of grayscale level. When « l^""!"' 
hS determined by Equation 23. exceeds a predetermined value range, a bjased W«^^'"i«^^"9 "^^ 
cmS It is necessary to provide such a maximum limitation value, without wh«h. sometmes the 
Si^v deteS ^^rect corresponding point because the block including a point at issue has accKlentally the 
SSti^ P^^aS a^e di2rS color. However, since the color «ference caused by camera char- 
TeS^iTSene^ not very Srge and therefore Is within a predetermined imitation range, an .ntroductK,n of 
such a Hmitatlon value would be useful arid practical. ., u. « 

With the biased block matching discontinued, normal block matching may be used to evalu^e «ie amil^ o^ 
viewfirSer images. Alternatively, the value derived from the biased btock n«t*ing nw '^^^'^^'j^^^ 
lll^Se^orJy at the upper limfted value of a correctable range 

puted witti the following equation. 

ei=zj:(ii-I2) *^2:I:(l1-l2))^/N+Nx^ 

wherein x=a:2:(l1-l2)/N!-T 

(4) In Step 13 of E.T*odimertt 1 (Determination of an Initial Position of a 

rno^S among different-time Frames t. f is further selected as a chamcteristic po.nt. In Embodment 3 add.- 
:SSX2^c«iidered for the setection. In Hg- 28. Frames 10 to 12 constitute different-time James to 
a^Sw^ by Camera 1. whUe Frames 20 to 22 constitute differait-time ft^ 

SSTsST^V sSehTRg 28 constitute same-time frames as each other. WhSe directing attertaon at PotrtP m 

iS^fS^tenSementbetweenlhediflerert-limeframesise^ 

nunber). and its mowemerit between the same-time frames with a vector Ba 

When set as described above, a poirt which meets the fonowing criterion wiB be selected as a cha^ 

i 

vector Bn is subslantiaBy consistent or moves sitostantiaUy consistently. 
^ ^iS^Sl^Z* Criterion (a), the fo«^ 

meets iie both criteria as a ehaiacterislic poinL ' 4 

fti^ vector An is siljstantiany consistent or nrwvessitostantially consistently ^ 

^ ^Srio^ i)^^wnds to the condition Introduced in Embodiment 1 As described above, when a ^ng 
with amum-eye camOT^em. it is possible to obtain depth infomiation from sanr«-time fram« only. For tte. ti » 
^^^^^a JZ* corresSoSSg relattonshi* between -e^finderimaQe^^obteinjgttie 
r^^ relationship, information obtainable from different-time frames l^^^^^^^^^^ ^.^SSL 
SS^feconsidered having been accurately traced, a point which simultaneously meets theabove *»cntena 

findw iimge. the towwn dynamic prograrnntiirig nwy be applied to obtain corie 

[Stage2] Aoqusilion of Depth bitormation 

Depth intom«lkxi is cakaialed based on the displacement of respective imas^^ 
StageTh n«lti-eye shooting, where the situatiori of F«. 1 3 is achieved at lime t depm ^ 

iha m^hrvj disclosed alStaoe 3 of Embocfimentl. ^ ^ 

^b^ JS^.^ .Bspective oam«as of the muHi-eye --e^.-y^-?:;^^ 
«hi»wahnne another assurned that the retetiooship ainong them and their magnffioation rales (a focal distonw 

SSiSi^i;^ (absol.;^«!S^ be Obtained, including ascatolactorKvvhichcannotbe deter- 
rrwied Bt Embodiment 1. 

[stages] Creafion of an Image 

An imaoe is created through the same process as that at Stage 4 in Embocftnenl 1 (Creation of an Iw) 

as m the above, a camera receives a stereo vi««nder imaoea^ «»oe 

far a 3-?^. Therefore, the viewfinder image captured by the camera will be ^f^^^^^^^ " 
Zs6on toSfad that a desired image can be created through image processma »iclud»ig an enhanced cfsptay. as 
descrl>ed in Eirixxliment 1 . 
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Embodiment 4. 

A technique for creating a viewfinder image seen from a different point, by utilizing a mouse and its dicWng action 
is descri>ed In Entxxliment 2. In the following, examples wHI be desaibed. where various viewfinder images seen from 
different points are created for a variety of purposes. ^ m 

As described above, accordng to the present invention, it is possWe to create a viewfinder image seen from a dif- 
ferent point without moving a camera. In this case, naturally, a viewfinder image seen from a viewpoint which is hypo- 
theticaOy located a shorter distance from the actual viewpoint will result in greater accuracy. By utiBzing this fact, the 
following applications can l>e achieved. 

1 . The creation of a viewfinder image with multi-viewpoints, based on a viewfinder image shot by a double-eye cam- 

When a stereo viewfinder image is available with a doid)le-eye camera system, a viewfinder image with mUti- 
viewpoints wiB t>e created by hypotheticaBy providing a third camera In other words, a point at which the hypothet- 
ical third camera is placed is detemined so that the third camera is set apart from the other cameras by a small 
space Then a viewTinder irnage seen from the thus deterrranedpoim is created. The t^ 
relatively accurate, and two viewfinder images actuafly c^tured by the two cameras of the doiWe-eye camera sys- 
tem are contiined together, so as to create a good viewTmder image with mufti-viewpoints. Si^sequently. additional 
considerations of depth infomiation would permit the creation display, which is seen from aiy 

20 of the mufti-viewpoints. 

2. Creal!ono*aViewfBiderlni«geinSlow^tolion 

The closest two d»ferenl4ime Frames in tenm of fcne are designated respectively as Frame t and i; viowpomte 
of ftarnes t andf are designated respecliveiy as Viewpoirt 
2S Viewpoint t to f from Frame ttof.no viewfinder image between mem 

thetical viewpoint betwewi Viewpoints t andW a viewfinder image seen from a different point, that is. a point 
between ViewpoWs t and r in this example, is newly aeated. A plurafity of viewffWer images seen from <f»erent 
points are created in tf«s way. Then, a secpjenlial dspUy of such viewfindw images woM present a viewfinder 
iniaoe in stow rnotiorv which has the loliowirKj efle^ 

^ a. The movemerrt among respective viewfinder infi^jes becomes smooth, instead of an origwiri ffekery rnove- 

b^With a smalls 

image between those frames is not degraded. 
35 c-NferiationinapathwhereaviewpoWniovesfromViewpoirtttorw 

finder image in slow motion. 

Ad(fitionaI considerations of depth infonnafion woirid pem« 
noted that the above mentioned technology can be fi^Jplied to san^ 

40 

Entxxfimert 5, which is substantially the same as Embodirnert 1 expect ^ 
aims to practice the foBowing image processing Using depth inform 

45 

1. Change in a Viewpoint 

Aocon^anymg the hypothetical change off a viewpoint a viewfWer m 
present invention when the viewpoirt is hypotheticaBy changed, a viewfin^ image seen from the changed vtewpom* 

so is cmtomaficaly created wtitle the camera is fceptfixBd. 

2. Partial Eiy ansi on or Con yressi on of m imagB 



By ulilzing depth Wormaioa the rnost natural and e^ 
& talscafing as revised. 
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3. Separation of an Image Area 

Pnr smaratina a desired image area, it is first necessary to fully recognize respective image areas. For an^ rec- 
nnJZ ^^S^^^T^pf<^e6. including a clustering method, txit they have had only ur«atslactory 
SlllS^^rjie^Jt^ntton permits an^urate area separatton in a totally different way from the convent«nal way. 

"''"linSSSS^on is obtained««ufl».*esameprac«s as that in Embodiment 1 . in thefc^lowing. only Stage 
4 (Creation of an Image) wW be descii»d,a5ltfiSB»l!om Errbodt^ 

[Stage 4] Creation of an Image 

A desired «ewfinder ima9eJ6a«««»ace«feH3to««*a«!«^Sn^^ obtained 
staa« « tTSTS* vlewfinder image frames have been demanded in extract»ig n^ssary ,nlom«tK,n. 

^ *is p^m adesired image based on only a single vlewfinder image frame at Stage 4. 

(1) A Viewfindei msageSfcen from a Different Point 

Ros 29 and 30 show a cor«apondinB »BJa6ons»*»b«ween an onspnal viewfihder image and one reKreated so as 
JT^ J^^r^r^ ^^Tpia 29 is the orianai viewfifiSer image, showing a tree, a house, and a person. 

SS^sSS. The!^o^ to the bottom 

des^ ibed in Stage 3. By reversely following the processes at Stages 1 to 3. rt s possWe to <^npute 
i^Xct«^8creen.basedonthishypothelical3^ma^oftheol^soa^^ 
srwwn^^ 30- Since no room is leflforaibitiBriness in*^ 

'"TSfewIilte PreteraUe to oonsiderand reflect a masWng relatiorship in creating an imaga Concret^^peak- 
ina EF?3oKLmf-e.acconpanyinflachanoeoftheviewpoint.ft^ 

S^^l, J« W^eSTto^^ a natural viewf Inder image, the bottom part c^ttietree should be co^ed 
bv the imaoe data of the house. In actual software processing, the creation should be started with an image part hawng 
aC^^ShToSrto^^e a natural viewllndWimage. Alternatively, the Z-buffer technique, which te^wKlely-«^ 
fn oS^,S?gU)hics. can be used tor this purpose. For obtaining a maskinB ^e^^ti^^P «™lfV^^ 
^^S^ftS Sade as to whether the sight vectors drecled at th^ 
J^^JLS rSJe another. When theSw vectors to Part^ 
SiSJtoSvLpoirt than Part B. it is known that P»t A Should be 

based on the thus computed intormalioa 

(2) Parliai Scafing of VI Image 

in an enhanced dteplay. one of the image <isplay techniques, a closer c^^^ 
wWteadlstart object may be made e«n more distant so that the contrast .ndeplh»OT^^ 

Fbr ttis image processing, aooordkig to the present invention, images are P«^f«^9«i*^,^*f*2 

depTSI^xmalim F^ll^ 29- '^'^J^'rl^li:;^:^t^-^ 

of tt^ oerson being expanded, the person haring the shortest depth among all obiects mthe drawing, thepersonmper- 
f^^^!Il!?;SSWto the'^.XTiesult. an effective enhanced display is achi««l. In this case, prefera- 
bly, a ^nasWng relationship is also lell^ .. ^ , . ^„,^„.,:^ ^ 

bM»uBe that area caii be expanded wiwit a pibbiem until it b pereeivBd as hawng nodaplh atalL Howwer «mi^ 

be pef«iyed as being ctoser than *»peTO the niagnify^ 
S^'^SSmsultW. unnatural vie«finder imaoa In the 

the presert invention. H is possWe to make coiidrtions such as that only areas with the sho^ 
SS^only ones with tt^est depth shoukJ be reduced, so as to create a natural a^ 

image in oompianoe with the laws of nafttf 
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In the abcwe, methods have bem described for creating a natural image, though unnatural images may sometinnes 
be demanded, such as when the sense ot unnaturalness needs to be emphasized with a larger display of a distant part 
than that of a doser part. Such an unnatural image may be used for games or the like. In any case, according to the 
present invention, the naturalness or unnaturalness can be freely created as desired. Conventionally, a natural Image 
may or may not have been created as a resuft of an accident where the scale of some parts of the image were changed. 
However, according to the present invention, the aeation of a natural or an unnatural image is ensured as requested. 

Once a natural image is created, in order to further carry out the above described process (1) or a process (3) to 
be described later on the natural image created, it is preferable to begin the process by changing the depth of the 
expanded or compressed areas. For example, when an area is doubled in size, its depth should be halved. Contrary, 
when an area is halved in size, its depth should be dodbled. This conrecUon is necessary because the size of an area 
is ffversely proportional to its depth. An image so conrected would ensure a natural image to be produced in the sub- 
sequent processes. ^ 
In conducting the image processing of the above (1) and (2). an image may be finalized by smoothing uneven parts 
along the edge of the image. For example, when re-creating the viewfinder image in Fig. 20 into that in Fig. 30. ft wUI 
never happen that all image parts in Fig. 29 con^e^xxxl to those in Fig. 30 with a one-byH)ne con^esponcfing relation- 
ship. Conaetely speaWng. since the space shown at the top right comer of me image m Fig. 30 may shew cbjects not 
seen at the same region of the image in F^j. 29. Therelore. in a naive creation of a viewfinder im^ of Fig. 30 based 
on that of Fig. 29. an image part to be shown at the re$pon is actually broken off in Fig. 30. This break-df causes a 
recess wHh respect to the ideal edge line of the imaga For the canr>e reason, all image parts included in Fig. 29 are not 
20 shown within the kleal edge of the BTiage in Fig. 30, v»*hilesonwim^ 

In Older to solve this problem and to maintain the original screen shape (rectangular in Ms example), such 
recesses are fffled with extra pixels. wKte such projectioiK are cut ^ 

with pixels having the same cokx as that of images in the adiacent region. When the acx>#o image piocessing (2) 
causes^terunevenness^ong the edge line of the amendmertwoiid solve the problem. 

25 amencknmt the edge lir»aff the image is displayed natml. 

(3) Separation of frrage 

Adesred image part is separated to be intivklually processed. Referring to Fig. 29. ft is assumed thai tfi e person, 
thehouseandthetreerespecthfdyhaveadeplhof 3m. 10m. and20^ 

is made such as th^ %«thin fiv©^m^ before starting the detection and judgencrt of t^^ 

respective parts. In Older to separate the house, the coTKitkxi may in 

"^ig. 32 is a viewfinder image created with the house separated from Fig. 29. After separating a desired image area, 
35 the rest may be left WanK or the separated area may be pasted on a differertviewfind^ 

As described above, the present invention provk*es methods for image recognitton and processirig Con- 
ventionaflu iinage areas have been separated manually or by means of a dustering me^ 
inventton provides a method for achie«ng an accurate area recogniti^ 

method, using depth information. . ^ . , 

40 Asdescrl3edthusfar.1hepresertinN«rttondisctosesairi^ 

mation. Since the series of processes can be li«y automated with software, the present nwrtion can be appbc^e 
owerawiderrsviga 

Embodiment 6. 

An appropriate apparatus in practicing Eirtjodi^iSn*^ will be descrijed. whkdi is,sU>^BjaM» ttie^8awe as that 
described in Entxxfimenia exceptthai itouj^B^e^type of image, instead of two types ohm^«i?^i% mia^ 
lor right and left eyes in En*x>dirnert 2: The^peSpriw^^ next be descnbed only in oonnecliCKnjnIh f^^Ji^ 
ences from Eittiocf man 2. A camera shoots anaiertso as to capture its viewfinder image, whdi « 8i«iied« tie 

so image* in»t drcuft 20 lobe stored in the ftai»fiin«rr24. A plurality of frames on aviewfa^<maQean^i|out 
fiomlKefiimem«norya*1»be processed thrtj^^ point detection araAfifi and the motion delKS- 

tioo circuit ao^teoWainingdeplhiriid^^ . 

SiibsaBB*iiKit«^^«i»06 craa^ ^ * viewfinder mage seen ««« 

point aoc±idamdiieiie mi mMlBiS^^ uvMcaoKSA. 

55 !«rious processes wiabecaiTied out incfarfngih e qeaien of a viewfinder image seen from a dfterent port, eaqpan- 
sioa conyre s s i on or sepaiafica asdescrfl>ed in EiitoKiment 5. 
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fmhodiment 7. 



A methnri for creatino an image for a 2-D display when receiving a stereo viewf inder image will be descrtoed 
d^re^ce E^S,^ 5 anr7 Is the same as that between Embodiment 1 and 3. Acoord.ng to 

EmbSrmer7 i??s gtTaj; ^ to obtain depth information with a high accuracy, and thereby achieve a h,ghly 
accurate creation of a viewf inder image seen from a different point as a final image. 

f =mhodiment 8. 

Similarly to Entxxliment 1. a method for displaying a good stereoscopic *™Sie^b^ on infcxn^ 
and 2^5 a 2 D umge. wUI be described. Embodiment 8 is different from Embodiment 1 .n that rt consider condrtions 

and ^S^Si^wbiSVbased on a 2-D image. That Is. tt« sender transmits televlston vnage signals (2-D jmage 
S?nf S^^LS^SXlh information. The receiver, on the other hand, divides the received mage s«nals«itoN«, 

r^p^mageparts 
S^^^i^S^S^^STright and len eye i^^^ 
«o««rt«re«imaaediSDlay apparatus, to achieve reproduction of a stereoscopic image 

in rtfe^SS^^DOonsider the nature of a parallax. In other word, as has already^ 

m« ^1 ?^ a ^^StoTtesed on an angular dHlerence between sight vectors, the extent of wh«h >«n^ among 
SX^^rS^^ disiScmg the same number Of pix^s. the P^^lf 
aispi^appw- Hfeniflv amMlus. Even assunvng that the size is the same, the parallax stall vanes 

5^"° ^ ^ viewer. Therefore, in order to enfoody the optimum 

^IT^Sxrfment 8 the correction value unique to each stereo image display apparatus is introducedji 

d J?i^SSi^F« 2^ S^^a^mcture of a stereo image display apparatus according to Entxxtanenta |nwhK*i 

J?SSST^dS.S.?^ares^»Bed viaanlrputterminal m 

ISf^^^^^T!™ Stecement droua 106. The buffer memory absorbs a deteyewsBd.m displace- 

rwTriSSv nanel 110 dfe^ysan irmge given a dteplacement in the displacement circuit 106. _ ^ ^ 

^ ^S^S^ is ini^di^ement circuit 106 determines the extent of displacement wrthrej- 

erenIfto^SrSr<S2^S^ but also toners unique to the apparatus. P--;^^ 
frlTTOM ^?2^e ROM 1 1 2 stores the optimum a)rrection value far t^ 

lowffig general rules. 

Retetina to the size of a (fisplay panel: tor a smaler display panel, a larger value is stored. 



value is stored. 



The dsplay circuit 1 06 gives a larger displacement » a d^ issTi^er o^^ 
value beingprUlermined accordUig to the above mentioned liies. As a result the opto^ 
achieved, which reflects the conditioiw unk^je to the display apparatus. 

Enixxliment 8 may also be applied wHh the fatowing technical vanations. 

1 AS shown in F«. 33. a volume 114 mayt»pr«nded far manually changing the exterrto^ 
supplementary aclustmert or an adjustatiert according to a personal preference can be perf^^ 



2. Disijiaeement may be flwen to bo* riji^ and left i 

3. As is descri>ed in the toreodng Nfckei Eleelmnics hia 444. a 3-D 

4. The ROI»« 112 may ha« preyed thereinaplumlrty of oorreclfan values, so astobe select 

tng on the situation. 

5 Correelfan^lues may be Stored in two one being changed dep^ 

beinochahoed<lB^ii«l*onlhe<isl^ 
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6 In the description atwe. the value of depth information is accurately proportional to the distance between the 
shooting position and the object. However, depth information may show the absolute distance between the shoot- 
ing position and the object. 

A sinple case will be taken as an exanple. where depth information comprises only three distinctions: that is. large, 
mecfium and small. When depth information indicates large" or a long distance, no displacement is made to cause no 
parallax to the image part When "medium" or a medium in^ance is indicated, some displacement is made to cause a 
parallax to the image part When "smafl" or a small cfistanoe is indicated, a large cBsplacement is made to cause a large 
paraOax to the image part . 
The thus sinpWied depth infc>rmation could reAice the vdume of transmission data for bro^tesh 

£K:hieve a circuit having a simple stnjcture for a stereo image dsplay appa r a tu s 
damns 

1. A method lor creating an ini^ lor a 3-0 display, conpnsing: 

a step <rf ejdracting depth information from a 2-D motion image: and 

a step of creating an image for a 3-D dspliqr acoonfeig to the depth information. 

2. A method for creating an image for a 3-0 cSsplay acoordir^ to daim 1 . wherein the step of extract»»g deplh mlor- 
mation connprises: 

a step of detecting a nwvemerrt of the 2-D nwtoi image; 

a step of calculating a relative 34> moMement between a scwe and a shooting viewpoint of t»e 2-D motion 
image; and 

a step of orfculatBig retative cfistwwes from tteahooting viewpoint to respective parts m 3-D space, which 
projected m 2-D motion inroge. based on the relative 3-D movenrienl and the 2-D m 

Imageperts. 

3w A mrtKXffbrcalctte&Ki depth ii iforortio i i^ t^^ nwtion irnage. corBy«tftO- 

a step of selecting tM tames with an iepeimpe^^ large movement l)ehween them from tie 2-D motion 

mge;an(^ . - ^ ""^ 

a step of cacuteiting the depih two frames. 

4. A mettxxj for calcutating depth rilormation accoitfing to daim 3, further comprising: 
a step of providng a plurality of representative points in a reference frame: 
a step of ctetemirtng a pliffality of corresponcfing points in a^ 

representative points: and 

a step ol^btaifwig a positioned relationsfqp between the representative and conrespondmg poans. 

wherein , , . .. 

a position d a con-esponcfing poM in anottier tame is predicted accordmg to the posftoral r^ionshp 
b^weenthe rejiresertative €Bid correspondmg pointe, 
manotfierfFEmna 

5l A method tor calculating depth ffiforrnation aooonftigito claim 3, further ujiiipi&ing: 

a step of pfpvicfing a phrality of representative poinlB in a reference frame; 

a step of delerffining a plur^ of corresponcSng poffits in another frame, so as to correspond to each ol 
representative'PQirits;^mt . ^^^ f^- - „ ■ 

a step <rf obtewwig a positional relafiorwh^p between the repr^^ 

each prWVWm^i^tative points bdng^daseffied eaher as a characteristic or a non-chwacterislic point; 



wfwin rnoie thm^a^^fodS^^ points mcwe tae hw oe n the reference mi other 

tames, such that a tbStf c***te^^ predetermined value. « is judgedihat^niowemertbelween 

ttie reference appreprSalei^^^^ the reference and ttie otfwr tames are ther^ve 



A mrthbd for calctteftig deptft ii li ui ui atio n^KXOitfng to daim 3. fitfther comprising: 
a step of providing a phrafity of representetive poinftB in a reference frame : 
a step of drternriring a plurality otf corresponcfeig points in anottier fra^ 
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Ihe representative points being dassified either as a characteristic or a non-characteristic point. 

iSSrillorethan a predetermined nun*er of the characteristic points move between the retoence and other 
IramestS.^ a^Scrol their movements exceeds a predetermined value, it is iudg«J tf«t a rr»vemerrt 
b^i n ?S reference and other frames is appropriately large, and the reference and other frames are therefore 
selected. 

T A method for calculating depth information according to any one of claims 3 to 6. wherein 

7. A "^'^l^l^toi^ thedeplh infom«tion is discontinued when the two frames with an approprately large 
movement between them cannot be selected from the 2-D motion image. 

s A method for calculating depth information according to any one of claims 5 to 7. wherein ^.^^ 

8. A "^^^^^^a a representative point relating to an image area having a geom^nc charactenstc. 
is adjusted tobeposifcned such that an Image area related thereto retain the geometnc charactersHc. 

9. A inethod for calculating depth information according to daim 8. wherein 

the image area having a geometric characteristic is an area indudmg a straight line. 

10. A method for caloteting depth intonrotion, comprising: 

a Bterirfixovkfing a pluraHy of representativB points in a reference franw: 

a ^Slion simuarity of Images between an image area Induding^pedhc pojnte 

v^ich ie%!bit«iy set miiother fnune. and a nea*y image area including the representatwe pants in the refer- 

a stwolewluatinarelalivepositkxial acceptability among the spec^ _^ ^^^.^^ 

a^ rt^SSno the sped^^ 

'"'"1:^ep'SS:nS^^Wabes. poim Where brt^ 
m^flna^^fol tSSJwidingpoints w»^ 

of calculating the depth infomation according to a positional relationship between the represer^afe^ 
points ^^e SrwSing pSte. the conesponding points having been determined through a senes of the 
above steps. 

11 A inrthod for ealculaling depth Mormalionaccoiding to daim 10. wherein ,^^„^ 

^SerttSSSiandpoUonalc^ 

thereoltl^^SiJX^SvS^uS^^ differential equation ind«ates a condition where a con*«ned 
value of both of the ei«lualions is an exlremum. 

12. A method for calculaling depth information according to any one crt claims 10 and ll.w^ ^^^^^ 
12. '^'^^^So?si3Lyofimagesisconductedbybiasedblo*ma 
evaluaSSbelSIUt when the blod« induding 1t«i identical o^^ 



and 



relative positional aoceptabBly of images is evaluated using the function of distance between neiilitwinfl 

cf tie above evaluations aiB treated in terms of distance in the oo^ 
respec^^^^^«!^ combined together and can be used *^ 

ing points. 

13. A m^hod for catodafing dapth Wormafion aooonfing to daim 12. wheran ^ .^..^ 

13. A """^'J^j^j^ 52^^ ofeimaBas l?«>ndud«J by biased bio* matdung wrthm the limrtad coaecion 

range, which has been pr»<lulaininad. 

14. A method for cakailafing depth inlormalion.coitiprising: 

a step of providing a plurality of representative points in a relerenceframe. 

a^jOtdrternininQaplindityolcoi^^ 
represerttiiiBpbinls; I 
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a step of obtaining a positional relationship between at least a characteristic point among the representative 
points arxl its corresponding point 

a point whose position moves steadily among a plurairty of frames shot at different times is selected as the 
characteristic point. 

15. A method for calculatirig depth inforrnation, conrprisffig: 

a step of provicfing a plurafity of representative points in a reference frame; 

a step of d^ermining a plirality <rf correspontfng points in another frame, so as e^ 

of tie representative points; and 

a step of obtaining a positwnal retetionshp beN»e^ 
points and Its correspondffig poM. 

wherein , ^ ^ . 

a point is selected as the characteristic point, whose displacement is substantially cof»«tent between 
frsnes simultaneously shot, and further si±»s^aly consistent or changes siA)StantiaIly correistently tn between 
oilier frames simuKaneously shot at a dose but different tmne. 

16. A method for calculaling depth information, comprisinQ: 

astepof providing a pfcffafity of representative points in a reference iniage; 

a step of detemining a plurafty tiff corresponcfing poorHs in another image, so as to correspond to each off 
the representative poinlC-respectivily; 

a step of obta^wig «i)06itional relationship bet^ 

a step of diculatBig depth in foi nn a tio n accowing to the positional reiationshp, 
wherein 

the cakailation <rf ttedepm «formation« tfsoo^ 
teristic points are selected from the representative pointe. 

17. A method for catouiatino d ^ in forma t io n acconSnrto«teim 16. virherein « 

tt» represeritativ© and oorrespondHig poofitB are respectively p 
2-0 motion me(^ 

18. A method for calcutetting depth information, based oi two frarnes 

the calculalion of thedeptti information te <iscontinued when nrwvOTient is small between the two frames. 

19. A method for calculating depth information off a 2-0 image, 

wherein 

when a deptii off any port ma certam mage is calculated as negati^ the depth is interpolated by usng 
depth infornr«tion off points closrty havfing positive del* valuer 

2a A method for ffnage processing usffig depth Wonnatioa . 

a step off creating a stereo mnage by giving a paraBax to a 2-D image according to its depth nfarmalion, 

where^ . , 

the paraBax is tran*rmed so as to laB w«wi a predelermbied range, so that ^ 

accordarig to the tansformed pmiax. 
2t. A method for ima^ prodwa^ to c*aim20^ 

the paiBflakls fineeftly compressed » the predetermined range having a point aibdianly 

d^srmmed w a nrvdcte value cl the predeternmed I 



22. Ainelhod for inMioeipi^^ 

wffierein ■ .r. . ^ ■ 

the paraB»«l|he1jg^^ transformed to be a closer v^ue off ettier an m>er or 

a lower rmtation vafcie ol tfo i^ 

23. A metixid for image processing aooorcfing to daim 20. 

wffierein 

^ paialw is non*»e»ty transfornwd such^iaft^^ vetfue s smooth ly converge to upper and km/er 
lmilrfbnvalues^D*t»^ 
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24. A method for image processing using depth information, comprising. i^,„«„i«„ 

a step of creating a stereo image by giving a parallax to a 2-D image according to its depth .nformaton. 

wherein 

the parallax is variable, wrtch is originally determined according to the depth infonnation. 

5S A method for image processing using depth information, comprising: 

25. A method^^ .maj^P^ ^ ^^9^ ^.^^ ^ ^ .^^^ ^^^.^^ ^ .nformaton; and 

a step of displaying the stereo inr^ge on a stereo image display apparatus. 

rS^^ to be conducted on the 2.D image so as to cause th^ 
condition inique to the stereo image display apparatus. 

26- A method for image processing according to daim 25. 
wherein 

the display condilion is determined according to a size of a display screen of the stereo image display appa- 
ratus and an assumed dfetance from the display screen to a viewer, and 

^^.SnTbe conducted on the 2-D image so as to cause a desired parallax is determmed .ndwdually 
based on the <Ssptey condition thus determined. 

27 A method for image processing using depth information, conrpristng. 

of <»Mtil^rSereo image tvgi^ 
depth information, 
wtierein 

an uneven iiTUKje frame oulBne caused by the paraOax tMiich has been given IS coo^ 

otL A method for imaoe processing using depth HTformatkxi, comprising, 

28. '^'^J^;;;^^^^^e^i;;^ge by givingaparayaxfor every i^^^ 

depth informatioa i. 

wherein 

a desired shape of an image frame is achieved by culBr^g oH a peripheral part of the image. 

29. A method for conducting an image processing to a 2-D image aeconJing to its d^ 

wherein 

an image area sUOject to the image processing is determined, based on the depth information. 

30. A method for image processing according to daim 29. 

wherein 

the image processirig is a process for chariging a size of the inriage area. 

ai. A method for image processing accofding to damn 30. 

the process far changlngasize of the image area isapcocess tor expandiriga^of an hnage area wrtha 
smaller depth so as to be relafively larger than a size of an image area with a larger depth. 

32. A method for image processing according to daim 29. 

11^^ processing is a process for conducting separabon of a desired image area. 

33. A method for image processmg according to daim 32. 

^J^Lralion is conducted with respect to an image area having a depth within a p«^^ 



34. A melfwd for irnage piacassing auujiiJiiig to daim 32. 

wherein _ 
an image area which has been separated is cortbined with another image. 

35. Amethod for conducting the image processing with respedtoa2-D image accoidingtoilsdepthinforn 

wlwrein 
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irages with viewpoints at a plurality of points on a hypothetical moving path, wh^e a shooting point of the 
2-D image is hypothetically moved, are created for use as a slow motion image, based on the depth infonmation. 
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< VIEWFINDER IMAGE SCENE> 
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<REPRODUCED RIGHT EYE IMAGE> 
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STAGE 4: CREATION OF IMAGE 
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Fig. 19 
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